Summary
Our client is a major player in the networking and telecom market that provides IP phones and other networking devices like routers and switches. With the goal of accelerating their testing services, they partnered with us to implement auto-generation of the test scripts to reduce manual effort and speed up the time-to-market. Their existing testing automation framework demands manual writing of test scripts, which vary in quality, demand trained testers and are challenging to capture all the various scenarios. Advancement in GenAI technology provides visibility into automating the test script generation process and addressing our client’s testing challenges.
The Challenge
Manual debugging and human intervention, leading to drops in efficiency
- Manual script development: Trained testers are required to manually understand test cases and develop Python scripts using framework APIs.
- Review and debugging: Periodic and rigorous reviews of test scripts, along with debugging efforts, are essential for maintaining reliable testing automation.
- Customization complexity: The framework demands the development of configuration files, parser functions and customized APIs, adding to the complexity of the testing process.
- Dependency on human expertise: Current practices heavily rely on the expertise of manual testers to anticipate various test conditions, leading to a dependency on human understanding and interpretation.
- Resource-intensive configuration: The framework's demand for configuration files and customized APIs add a resource-intensive layer, requiring additional development efforts and time.
- Knowledge and instruction requirements: While GenAI can automate script generation, there is a challenge in providing carefully crafted knowledge and instructions to ensure comprehensive coverage, minimal reviews and effective adaptation to configuration changes.
The Objective
Navigating testing challenges with GenAI, and transforming complexity into automation
We emerged with a strategy for auto-generation to address the client’s challenges as follows:
- Automated test script generation from test cases as well as requirements
- Use of test script template to standardize the generated test scripts
- Support to generate custom APIs in the provided Python-based testing framework
- Multiple framework integration for traffic simulation in network device testing
The Solution
HCLTech AI Force — Automated test script generation
To address customer demand, HCLTech AI Force brings an automated test script generation use case in the client-preferred Python-based automation testing framework.
- It has an orchestration framework that crafts instruction to the generative AI models. It uses the client-preferred script template, test cases, configurations, API descriptions and other related external knowledge sources to understand the preferences and scenarios for test script generation.
- It also generates a textual summary of the generated test script for easy understanding.
- It provides external connectors to export the test scripts to test management tools and a download option to get the test script files. A test report is also generated describing the summary, generation cost (if applicable) and sample of generated test scripts.
- It also supports secured online generative AI models hosted in cloud servers and local AI models that can run on the client’s premises. It enables the client to control their data consumption through these AI services.
The Impact
With the use of this solution in the client testing automation service, we empowered them to achieve reliable outcomes:
- Approximately 40% reduction in bandwidth usage realized in testing services
- The quality of test scripts has been improved with precise comments/descriptions to ensure script standard and readability
- There was a significant reduction in debugging time with zero syntax error
- Accelerated testing services with faster script generation, minimized review time and minor script revisions
